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ABSTRACT 

Despite  major  advances  in  autonomous  vehicle  technologies,  human- 
controlled  ROVs  (remotely-operated  vehicles)  continue  to  fill  an  important  role  in 
underwater  work.  To  perform  effectively,  however,  the  human  operator  requires 
meaningful  cues  for  spatial  orientation,  good  workspace  visibility,  and  tight 
feedback  about  manipulator  behavior.  These  needs  can  be  hard  to  support  in 
actual  undersea  operations.  Telerobot  designers  for  space  missions  have 
addressed  these  challenges  by  presenting  a  graphic,  virtual  reality  model  of  the 
workspace  to  the  operator,  who  then  performs  tasks  on  this  representation  of  the 
actual  work  site.  Real-time  graphic  modeling  can  (1)  maintain  a  continuous,  clear 
depiction  of  the  workspace  that  is  largely  independent  of  communications 
bandwidth,  (2)  allow  arbitrary  shading  and  perspective  of  the  workspace,  (3) 
provide  integrated  navigation  and  orienting  cues,  and  (4)  support  a  rich,  multi- 
sensory  feedback  environment. 

The  use  of  virtual  reality  technologies  for  operator  interface  design  is  being 
investigated  at  NCCOSC  for  undersea  ROV  applications.  A  general-purpose 
virtual  reality  testbed  is  described  which  involves  a  dedicated  virtual  reality 
system  for  underwater  applications,  together  with  a  manipulator  system  and 
supporting  software.  The  objectives  of  the  testbed  are  to  examine  fundamental 
human  performance  and  engineering  issues  connected  with  operating  on  a 
virtual  workspace  for  real  telerobotic  tasks,  and  to  benchmark  emerging 
telerobotic  technologies  in  a  standardized  test  environment. 


INTRODUCTION 

The  U.S.  Navy  performs  a  wide  range  of  undersea  missions  including  deep 
water  search  and  recovery,  mine  detection  and  removal,  ship  servicing,  sensor 
placement,  and  support  for  scientific  research.  For  many  missions,  the  choice 
of  telerobotic  systems  over  human  divers  is  driven  by  operational  requirements 
(e.g.,  water  depth,  environmental  hazards,  etc.)  and  by  concern  for  human 
safety.  Unfortunately,  an  ROV  is  almost  never  as  effective  as  a  human  diver 
performing  the  same  work.  While  human  perception,  decision-making,  and 
manipulation  are  essential  for  most  underwater  tasks,  constrained  sensory 
feedback  from  the  work  site  (via  cable  or  acoustic  links)  limits  the  performance 
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that  can  be  currently  achieved  with  telerobotic  systems  and  provides  a  relatively 
poor  substitute  for  direct  human  presence  (e.g.,  Pepper,  1986). 

There  are  many  impediments  to  good  operator  feedback.  TV  cameras  and 
other  imaging  sensors,  for  example,  are  usually  mounted  at  fixed  points  on  the 
remote  vehicle,  with  restricted  fields-of-view  or  adjustment  ranges.  Operator 
perspective  on  the  environment  is  therefore  tied  to  whatever  "tunnel  view"  can  be 
achieved  through  the  sensors  by  vehicle  positioning  and  self-contained  lighting. 
Support  references  for  navigating  to  the  work  site,  orienting  to  the  workspace 
(i.e.,  vehicle  station-keeping),  and  manipulator  operation  are  gathered  through 
other  devices  and  displays  which  the  operator  must  mentally  integrate  into  a 
single  picture  of  the  immediate  environment.  A  representative  operator  station 
for  performing  these  tasks  is  shown  in  Figure  1 . 


In  addition,  many  tasks 
take  place  close  to  the 
ocean  bottom,  where 
sediment  is  stirred  up  by 
water  currents  or  work 
activities.  This  further 
limits  visibility  of  the 
workspace  and  reduces 
effectiveness  by  forcing 
divers  or  ROV  operators 
to  stop  work  until  the 
sediment  clears,  i.e.,  a 
"move  and  wait" 
strategy. 


Figure  1 .  Conventional  telerobotic  control  station 

Limited  operator  feedback  and  poor  interface  design  constrain  task 
productivity  and  extend  task  completion  times.  These  effects,  in  turn,  generate 
higher  costs  for  surface  support  resources. 


Mediated  Telerobotics:  the  Virtual  Reality  Interface 

Space  telerobotic  systems,  being  developed  for  satellite  servicing  missions, 
must  support  precise  manipulation.  This  requires  timely  and  accurate  operator 
feedback  about  the  remote  work  site,  communicated  with  a  limited-bandwidth, 
time-delayed  channel  (e.g.,  Sheridan,  1992).  Designers  of  such  systems  have 
utilized  graphical,  predictive  operator  interfaces  as  a  method  for  achieving  the 
necessary  feedback  support  with  considerable  success  (e.g.,  Schenker,  et  al, 

1 991 ;  Kim,  1 993).  The  operator  of  such  a  telerobot  acts  on  an  artificial  depiction 
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of  the  work  environment,  while  the  computing  subsystems  process  raw  sensor 
information  to  ensure  that  the  depiction  is  physically  matched  to  the  real 
environment.  The  operator  can  test  and  preview  the  effects  of  manipulator 
action  in  this  surrogate  environment  before  these  actions  are  physically 
executed.  Because  most  of  the  data  processing  occurs  locally,  i.e.,  at  the 
operator  station,  the  amount  of  command  information  that  is  actually  sent  to  the 
telerobot  site  is  minimized.  This  reduces  the  required  bandwidth  for 
communication  with  the  remote  vehicle. 

Telerobotic  interfaces  are  distinguished  from  the  larger  domain  of  virtual 
reality  systems  in  that  operator  actions  are  ultimately  realized  in  the  physical 
world.  Virtual  reality  is  nevertheless  an  integral  part  of  any  telerobotic  system,  in 
that  the  operator  is  physically  removed  from  the  work  setting  and  all  actions  are 
based  on  its  representation  (i.e.,  via  the  operator's  displays.  Virtual  reality 
techniques  have  been  successfully  used  to  support  real-time  operator 
performance  in  a  great  many  physical  environments  including  aircraft  piloting 
(e.g.,  Furness,  1986),  data  visualization  (e.g.,  Fisher  et.  al.,  1987),  and 
manipulation  of  scaled  physical  objects  (e.g..  Brooks,  1988),  as  well  as 
telerobotics  (e.g.,  Tachi  et.  al.,  1994). 

Virtual  reality  techniques  can  fuse  real  sensor  data  into  an  integrated 
depiction  of  the  remote  environment,  and  can  enhance  that  depiction  to  support 
operator  needs.  Examples  of  such  techniques  include: 


1 .  the  ability  to 
arbitrarily  establish 
lighting  and  shadowing 
to  support  the  best 
scene  visibility.  "Virtual 
lights"  can  be  created 
and  placed  without 
regard  for  the 
constraints  of  physical 
equipment  (Figure  2); 


Figure  2.  Virtual  reality  scene  with  and  without 
arbitrary  lighting  and  shading 

2.  the  ability  to  establish  arbitrary  viewpoints  of  the  work  area  (i.e.,  different 
viewing  angles  or  distances,  or  multiple  perspectives).  Because  all 
characteristics  of  the  display  are  computer-generated,  the  operator  controls 
"virtual  sensors"  that  are  independent  of  constraints  at  the  work  site,  such  as 
physical  obstructions  or  visual  interference  by  the  manipulator  itself  (Figure  3). 
This  can  provide  information  about  unseen  features  of  the  work  space  or  unseen 
consequences  of  manipulation; 
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Figure  3.  Multiple  views  of  manipulator  and  work  space  combined 
in  a  single  virtual  reality  interface 

3.  the  ability  to  generate  seamless,  360-degree  views  of  the  environment 
surrounding  the  work  site,  out  to  essentially  arbitrary  ranges.  Such  "virtual 
visibility"  offers  a  large-scale  context  that  can  aid  underwater  navigation  and 
search  tasks,  and  can  furnish  a  better  sense  of  vehicle  orientation  in  three- 
dimensions. 

4.  the  capability  to  provide  multi-sensory  operator  feedback,  e.g.,  via 
integrated  visual,  auditory,  and  haptic  displays.  Human  manipulation  skills  are 
almost  entirely  multi-sensory  in  nature  and  multi-sensory  display  has  historically 
been  a  central  thrust  of  virtual  reality  technology  development  (Burdea  and 
Coiffet,  1994). 


THE  NCCOSC  PROGRAM  IN  TELEROBOTICS  AND  VIRTUAL  REALITY 

The  significant  potential  of  virtual  reality  applications  in  telerobotic  design  is 
accompanied  by  an  equally  significant  set  of  development  issues  requiring 
research  and  engineering  attention.  NCCOSC  is  addressing  some  of  these 
issues  by  combining  its  experience  with  telerobot  development  (e.g.,  Shimamoto, 
1993)  and  virtual  reality  systems  (e.g.,  Murray,  1995)  in  a  new  research  effort. 
The  program  focuses  on  benchmarking  operator  performance  as  a  function  of 
changes  in  virtual  reality  model  characteristics.  A  unique,  virtual  reality-based 
testbed  facility  is  being  developed  to  support  these  investigations. 

Research  testbed  facility 

The  NCCOSC  test  bed  facility  consists  of  a  virtual  environment  interface  and 
control  system,  a  Western  Space  and  Marine,  Inc.  MK-37  remote  manipulator 
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(typical  of  many  systems  used  for  underwater  work)  and  a  hardware/software 
architecture  designed  to  support  both  in-house  and  collaborative  research 
efforts. 

The  hub  of  the  research  facility  consists  of  the  Virtual  Environment  for 
Undersea  Telepresence  (VEUTel),  a  system  developed  by  Innovation 
Associates,  Inc.  (Schebor,  1994)  under  a  Small  Business  Innovation  Research 
(SBIR)  project.  VEUTel  (Figure  4)  is  a  virtual  reality  interface  system  designed 
expressly  for  control  of  a  remote  underwater  telerobot.  VEUTel  features: 


Local  Site 


Figure  4.  VEUTel  system  components  and  architecture 

1 .  an  advanced  three-dimensional,  stereoscopic  display  of  the  remote 
telerobotic  work  space.  The  display  can  be  implemented  on  a  conventional  flat- 
panel  (CRT)  display  or  on  a  head-slaved  helmet-mounted  display  (FIMD); 

2.  dynamic  creation  and  updating  of  the  virtual  reality  model  using  an  integrated, 
vision-based  tracking  system  at  the  remote  vehicle  to  measure  and  correct 
deviations  between  the  simulated  and  real  environments; 

3.  an  advanced  human-machine  interface  employing  an  intuitive,  virtual  reality 
interface  with  multi-modal  user  interaction.  The  interface  supports  complete 
sensor  and  manipulator  control  through  a  conventional  telerobot  master  system, 
through  a  computer-linked  glove,  and  through  voice  command.  All  controls  and 
system  status  data  are  integrated  into  the  virtual  environment  itself,  so  the 
operator  can  access  any  required  information  in  a  single,  immersive  display; 

4.  a  flexible,  object-oriented  software  architecture  to  generate  remote  scene 
environments  and  objects,  and  to  accommodate  alternate  hardware 
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configurations  (e.g.,  alternative  manipulators,  arms,  or  control  dynamics).  Such 
flexibility  will  support  the  integration  of  tactile  sensors  into  later  phases  of  the 
research  program  without  major  changes  to  other  VEUTel  components. 


Research  program 

The  NCCOSC  research  program  seeks  to  develop  data  bases  of  operator 
performance  as  functions  of  interface  characteristics.  This  is  achieved  in 
phases,  by  first  measuring  performance  of  basic  operator  actions,  and  then 
moving  to  more  complex,  operationally-relevant  tasks.  Visual  interface  features 
are  examined  first,  followed  by  other  forms  of  operator  feedback,  particularly 
haptic  displays. 

Operator  performance  benchmarks  are  generated  through  a  standardized 
battery  of  perceptual-motor  tasks.  The  test  battery  involves  a  series  of  very 
elemental  actions  (e.g.,  move,  rotate,  grasp,  turn,  etc.),  which  are  individually 
measured  and  modeled  for  time  and  accuracy.  Performance  prediction  for  a 
complex  task  can  then  be  done  analytically,  by  breaking  the  complex  task  down 
into  its  constituent  elemental  actions  and  adding  together  the  appropriate  model 
data.  The  technique  has  an  extensive  history  in  human  work  performance 
measurement  and  its  use  as  a  human-machine  performance  tool  has  been 
advocated  elsewhere  (e.g..  Pepper  and  Kaomea,  1988).  Advantages  of  this 
approach  are:  (1)  the  test  battery  can  be  applied  across  different  telerobot  and 
display  configurations  for  system  comparisons,  (2)  the  use  of  elemental  actions 
can  help  to  identify  specific  engineering  deficiencies  in  the  human-machine 
system  (e.g.,  backlash  in  the  manipulator  arm  for  linear  movements  of  different 
lengths),  and  (3)  predictions  of  "real  world"  performance  can  be  made  for  novel 
tasks  that  may  never  have  been  performed  before. 

NCCOSC  is  located  adjacent  to  several  U.S.  Navy  communities  that  use 
ROVs  and  manned  submersibles  for  underwater  work.  Volunteer  operators  from 
these  communities  will  be  used  to  generate  the  performance  data  bases  for  the 
perceptual-motor  task  series.  Tests  are  conducted  using  direct  vision  as  the 
baseline,  i.e.,  where  the  operator  can  directly  view  the  work  space.  The  test 
battery  is  repeated  using  a  set  of  virtual  reality  depictions  with  differing 
characteristics  (e.g.,  with  and  without  control  of  scene  perspective,  with  and 
without  control  of  light  and  shade,  etc.).  Because  the  physical  work  space  and 
the  virtual  reality  depiction  are  both  implemented  in  the  laboratory,  model 
accuracy  and  performance  measurement  can  be  precisely  controlled. 

The  second  phase  of  the  NCCOSC  program  involves  testing  with  a  set  of 
operational  tasks,  defined  by  the  Navy  operators,  under  the  same  series  of 
display  conditions.  Empirical  performance  with  these  more  complex  tasks  will  be 
compared  to  predicted  performance  from  the  modeling  technique  described 
earlier.  If  the  results  are  in  sufficient  agreement,  then  two  objectives  will  be 
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achieved:  (1)  the  use  of  standardized  tasks  as  a  benchmarking  method  will  be 
validated.  If  such  validation  is  obtained,  then  the  methods  can  be  applied  across 
other  telerobotic  applications  and  systems,  providing  a  reliable  general-purpose 
tool  for  performance  characterization  and  prediction,  and  (2)  the  relative 
contribution  of  different  visual  display  features  (e.g.,  perspective  control,  lighting 
control,  etc.)  to  performance  support  can  be  evaluated  by  measuring  the 
sensitivity  of  task  performance  to  the  presence  or  absence  of  these  features. 
Such  data  can  be  used  to  model  different  telerobotic  systems  and  tasks. 


Research  collaboration 

NCCOSC  expects  to  introduce  haptic  feedback  and  other  multi-sensory 
displays  into  its  research  program  through  collaboration  with  other  laboratories, 
where  studies  of  haptic  performance  are  already  in  place  (e.g.,  Cutkosky  et.  al., 
1992).  Most  human  activities  are  multi-sensory  in  nature,  and  additional 
feedback  modes  can  supplement  the  visual  sense  in  important  ways.  Haptic 
feedback  is  a  logical  first  choice  for  multi-sensory  investigations  in  that  the  sense 
of  touch  and  feel  may  provide  the  primary  means  for  updating  the  underwater 
world  model  used  for  the  virtual  reality  interface,  especially  if  the  location, 
orientation,  or  condition  of  the  work  space  is  not  precisely  known  or  if  imaging 
sensors  (e.g.,  video  cameras,  ultrasound,  laser,  etc.)  cannot  provide  sufficient 
information.  It  has  been  demonstrated,  for  example,  that  a  visual  model  of  an 
unknown  object  can  be  built  up  solely  by  registering  the  physical  locations  of 
contact  events  from  a  telerobot  end  effector  (e.g..  Oriels  et  al,  1992;  Fyler,  1981). 

The  VEUTel  architecture  of  the  NCCOSC  research  testbed  can  be  easily 
reconfigured  to  add  haptic  feedback,  although  the  motor  performance  test 
battery  used  for  visual  displays  will  require  extensions  to  include  such  task¬ 
relevant  characteristics  as  slip  detection  and  grip  force.  The  fundamental  testing 
concept  is  similar,  however,  for  both  visual  and  haptic  feedback  modes. 


Further  approaches  to  operator  support 

Generating  an  immersive,  virtual  reality  scene  for  telerobotic  operations  is  not 
difficult.  Ensuring  that  the  scene  correctly  represents  the  physical  world, 
however,  and  that  operations  on  that  world  are  updated  in  a  timely  fashion  is  a 
formidable  challenge.  Moving  a  remote  manipulator  in  the  absence  of  accurate 
feedback  can  be  both  inefficient  and  dangerous.  It  falls  to  investigations  such  as 
those  described  here  to  determine  exactly  how  accurate  the  virtual  reality  scene 
needs  to  be,  and  how  rapidly  it  must  be  updated.  Once  such  (probably  task- 
dependent)  data  are  developed,  however,  many  interesting  support  tools  can  be 
added  to  the  operator's  display  to  assist  performance. 
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Using  an  alternate  application  of  virtual  reality,  the  displayed  scene  might  rely 
on  raw  imaging  with  graphical  overlays  to  provide  enhancements  for  recognizing 
task-critical  elements  of  the  work  space.  This  approach  has  been  taken  in  space 
applications  studies  for  satellite  repair  missions  (e.g.,  Bejczy  et.  al.,  1990;  Kim, 

1 993).  It  provides  the  operator  with  immediate  recognition  of  a  problem  when 
the  raw  (although  possibly  degraded  or  time-late)  scene  image  and  its  graphical 
overlay  do  not  match.  To  the  extent  that  the  raw  image  must  be  updated, 
communication  bandwidth  requirements  increase.  With  the  use  of  high 
bandwidth  fiber-optic  data  links  in  most  current  ROV  systems,  however,  this  is 
not  a  severe  penalty.  The  performance  gain  comes  from  the  design  of  the 
graphical  overlays,  which  can  contain  special  highlighting  information  to  guide 
operator  procedures,  provide  interactive  decision  support  to  the  operator,  or  offer 
hazard  warnings. 

In  addition  to  providing  a  clear  view  of  the  underwater  work  scene,  a  virtual 
reality  interface  can  add  elements  to  the  image  as  necessary,  to  guide  a  desired 
operator  behavior.  “Virtual  landmarks”  or  navigation  grids,  for  example,  with 
horizon  lines  or  highlighted  depictions  of  bottom  terrain  (from  existing  data  bases 
or  real-time  mapping  of  the  sea  floor)  could  help  the  operator  to  navigate  and 
orient  to  the  work  site.  On  a  smaller  scale,  visual  or  haptic  markers  could  be 
inserted  into  the  environment  as  guides  for  a  specific  manipulator  path,  to  help 
the  operator  locate  particular  regions  of  an  object,  or  as  barriers,  to  help  the 
operator  avoid  certain  sensitive  or  dangerous  regions  of  the  work  site  (e.g., 
Rosenberg,  1992;  Sayers  and  Paul,  1994).  The  underlying  theme  of  these  virtual 
reality  applications  is  that  the  relation  between  the  real  and  artificial  worlds  need 
not  be  a  one-to-one  mapping;  information  can  also  be  modified  or  enhanced  to 
obtain  a  desired  operator  performance. 


Summary 

NCCOSC  is  beginning  a  new  program  in  ROV  operator  interface 
development  that  focuses  on  immersive,  virtual  reality  methods  for  system 
control.  The  program  is  distinguished  from  similar  efforts  in  its  use  of  a 
reconfigurable  test  bed  (VEUTel)  and  a  standardized  benchmarking  method  for 
performance  measurement.  The  objectives  of  the  NCCOSC  effort  are  to 
develop  and  demonstrate  the  utility  of  a  virtual  reality  interface,  to  develop  time 
and  accuracy  performance  data  for  a  core  set  of  elemental  actions  as  functions 
of  virtual  reality  display  features  (e.g.,  perspective,  scale,  light  and  shadow, 
multiple  views,  etc.),  and  to  define  a  model  of  human  information  processing 
that  relates  these  display  features  to  skilled  task  performance. 


8 


REFERENCES 


Bejczy,  A.,  Kim,  W.,  and  Venema,  S.  (1990).  The  phantom  robot;  Predictive 
displays  for  teleoperation  with  time  delay.  In  Proceedings  of  the  1990 
International  Conference  on  Robotics  and  Automation,  Cincinnati,  OH,  May. 

Brooks,  F.  (1988).  Grasping  reality  through  illusion:  interactive  graphics  serving 
science.  In  Proceedings  of  SIGCHI-88,  May. 

Burdea,  G.  and  Coiffet,  P.  (1994).  Virtual  Reality  Technology.  New  York:  John 
Wiley  &  Sons,  Inc. 

Cutkosky,  M.,  Howe,  R.,  Johansson,  R.,  Westling,  G.,  and  Edin,  B.  (1992). 
Tactile  sensing  and  control  in  humans  and  robotic/teleoperated  systems.  Final 
report  (Contract  N0001490J4014).  Stanford  University  Department  of 
Mechanical  Engineering. 

Oriels,  M.,  Spain,  H.  and  Aviles,  W.  (1992).  The  effect  of  sensory  feedback  on 
the  identification  of  objects  using  a  teleoperator.  In  Automation  in  Construction 
1,  Amsterdam:  Elsevier. 

Fisher,  S.,  McGreevy,  M.,  Humphries,  J.  and  Robinett,  W.  (1987).  Virtual 
interface  environment  for  telepresence  applications.  In  J.  Berger  (Ed.), 
Proceedings  of  ANS  International  Topical  Meeting  on  Remote  systems  and 
Robotics  in  Hostile  Environments. 

Furness,  T.  (1986).  The  super  cockpit  and  its  human  factors  challenges.  In 
Proceedings  of  the  Annual  Meeting  of  the  Human  Factors  Society,  vol.  1 . 

Fyler,  D.  (1981).  Computer  graphic  representation  of  remote  environments  using 
position  tactile  sensors.  SM  Thesis,  Massachusetts  Institute  of  Technology, 
August. 

Kim,  W.  (1993).  Graphical  operator  interface  for  space  telerobotics.  In 
Proceedings  of  IEEE  International  Conference  on  Robotics  and  Automation, 
Atlanta,  GA. 

Murray,  S.  (1995).  The  NCCOSC  Virtual  Presence  Program.  HCI  International, 
Yokohama,  Japan,  July. 

Pepper,  R.  (1986).  Human  factors  in  remote  vehicle  control.  In  Proceedings  of 
the  Annual  Meeting  of  the  Human  Factors  Society.  Santa  Monica:  HFS. 

Pepper,  R.  and  Kaomea,  P.  (1988).  Teleoperation:  Telepresence  and 
performance  assessment.  Presented  at  the  International  Ergonomics  Society 
Annual  Meeting,  Bristol,  UK. 


9 


Rosenberg,  L.  (1992).  The  use  of  virtual  fixtures  as  perceptual  overlays  to 
enhance  operator  performance  in  remote  environments.  Technical  Report, 
Center  for  Design  Research,  Stanford  University,  September. 

Sayers,  C.  and  Paul,  R.  (1994).  An  operator  interface  for  teleprogramming 
employing  synthetic  fixtures.  Presence,  3(4). 

Schebor,  F.  (1994).  VEUTel:  A  virtual  environment  for  undersea  telepresence. 
Technical  Operating  Report,  Innovation  Associates  (Navy  Contract:  N66001-94- 
C-7010). 

Schenker,  P.S.,  Bejczy,  A.T.,  Kim,  W.S.  and  Lee,  S.  (1991).  Advanced  man- 
machine  interfaces  and  control  architecture  for  dexterous  teleoperations,  hi 
Proceedings  of  IEEE  Oceans  '91 ,  Honolulu,  HI. 

Sheridan,  T.B.  (1992).  Telerobotics,  automation,  and  human  supervisory  control. 
Cambridge,  MA:  MIT  Press. 

Shimamoto,  M.  (1993).  Teleoperator/telepresence  system  (TOPS)  concept 
verification  model  (CVM)  development.  In  N.  Saxeda  (Ed.),  Recent  Advances  in 
Marine  Science  and  Technology,  92,  pp.  97  -  104.  Honolulu:  PACON 
International. 

Tachi,  S.  and  Yasuda,  K.  (1994).  Evaluation  experiments  of  a  teleexistence 
manipulation  system.  Presence,  3(1). 


10 


